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Barret, 



Described is an interlocking scheme which permits 
multiprocessing in a shared storage configuration with 
each central processing unit (CPU) having a private 
high-speed buffer storage utilizing the store-in-buffer 
concept. The basic problem solved is insuring that all 
processors access the latest copy of common data with 
minimum performance impact. The system allows 
fetch-only copies of the same shared storage block to 
exist simultaneously in all private storages, but only 
one private store is allowed to contain a block of data 
currently being stored into. Disclosed, in addition to 
the normal controls necessary to search a high speed 
buffer to determine whether or not the data required 
by the processor is in the buffer, is means for intercon- 
necting the processors sharing a main storage. The in- 
terconnection is for broadcasting address information 
from one processor to the storage control mechanism 
of other processors for the purpose of invalidating 
data in other private storages or insuring that data ob- 
tained by one processor from the shared storage is the 
most current value. That is, if data has been modified 
in the buffer of another processor, that data must be 
returned to the shared storage in its modified form to 
insure that the one processor receives the most cur- 
rent data. 

8 Claims, 4 Drawing Figures 



SHARED MAIN STORAGE 



15- 



a_3T 



PRIVATE 
ST0RACE (A I 



21 



STORAGE 
CONTROL 



DIRECTORY 



PROCESSOR A 



3t 50 



^21 i2 W 



J / 1 

?9 



STQfUCE 
C0ITI01 



DIRECTORY 

r 



i 



PRIVATE 
ST0R4CE (8) 



- 18 



PROCESSOR B 



10/09/2003, EAST Version: 1.04.0000 



PATENTELM22BW 3.735,360 

SHEET 1 OF 3 



FIG. 1 




.16 



PROCESSOR A 



PROCESSOR B 



INVENTORS 
DAVID W. ANDERSON 
RICHARD N. GUSIAFSON 
LANCE H. JOHNSON 
gy FRANCIS J. SPARACIO 

ATTORNEY 



10/09/2003, EAST Version: 1.04.0000 



PATENTED M 2 2 1973 



3.735,360 



SHEET 2 OF 3 



FETCH /STORE REQUEST 



63 



64 



(y XFETCH REQUEST 



FIG. 2 



BLOCK VALID OH FOR II- 
BUf FED II) 



,71 



STORE ^66 



PICK BLOCK TO BE REPLACED IN 
BUFFER (A) (CALL IT 8-2) 



FETCH ONLY ON FOR 
IN BUFFER (A) 



BROADCAST 



IS BLOCK VALIO ON FOR 
8-1 IN BUFFER (B) 




TURN OFF BLOCK << 
VALID FOR B-t IN 
BUFFER (B) 






67 

\ ' 


TURN OFF 
FETCH ONLY 
FOR B-1 IN 
BUFFER (A) 



.70 



<P<3 



BLOCK VALIO AND STORE ON FOR 
2 IN BUFFER (A) 



J 7? 

5*P 



WRITE 8-2 INTO MAIN MEMORY 
FROM BUFFER (A) TURN OFF 
STORE FOR B-2 IN BUFFER (A) 



BROADCAST 



^73 



6B 



1 



IS BLOCK VALIO ON FOR (B-1) 
IN BUFFER (B) ' 




STORE ON FOR (B-1) IN BUFFER (B)Y7 




76 



jL 



WRITE B-1 INTO MAIN 
MEMORY FROM BUFFER 
IB) TURN OFF STORE FOR 
B-t IN BUFFER (B) 



.77 



FETCH REQUEST 



79 



J. 



81 

J— 



TURN OFF FETCH 
ONLY FOR B-1 IN 
BUFFER (A) 



TURN ON FETCH ONLY 
FOR B-1 IN BUFFER (B) 



82 



sT>® 

STORE 1 



TURN OFF BLOCK 
VALIO FOR B-1 
IN BUFFER (8) 



TURN ON FETCH ONLY 
FOR 8-1 IN 91'fft'. (A' 



J- 



READ FROM MAIN NEMORY 
INTO B-1 OF BUFFER (A) 
TURN ON BLOCK VALIO FOR 
B-1 IN BUFFER (A) 

I 



78 



.83 



STORE 




jj X FETCH REQUEST ) >Q- 



NRITE DATA FROM CPU (A) 
INTO 8-1 OF BUFFER (A) 
TURN ON STORE FOR 6-1 
OF BUFFER (A) 



65 



READ DATA FROM B-1 OF 
BUFFER (A)-RETURN TO 
CPU (A) 



10/09/2003, EAST Version: 1.04.0000 



PATENTED 2 2 B73 



3,735.360 



SHEET 3 OF 3 



STORE 



ONLY COPt 



RESET REMOTE 
F BIT 



OR 



HOI 



STORE 



a 



-too 



TRANSFER BLOCK 



SET REMOTE 



FETCH 



pck r*r = 

TM-Z a 



BROADCAST A00 ^29 



REPLACE A00 ,49 



LOCAL A00 



BROAOCAST 



ADDRESS 



REMOTE 
'FETCH 1 



29 



a 



102 



103 



105, 



1C£ 



SET V BIT 



RESET F BIT 

FIG. 3 



OR U " 



OR 



FROM CPU 



51 



GT 



,52 



FROM SHARED STORAGE , 



GT 



GT Z, 



T 

36 



'86 



30 



FETCH /" 



STORE 



TRA NSrF T/ 46 . fpR~U 
block H — hf 



27 



38 



,50 

I 0 1 2 



IE 



COMPARE 

n 



-39 



a 



'106 



BLOCK 

RESTORE BLOCK 



BROADCAST ADO 



41 
42 



a 



44 



11 



a 



L87 



i 



51 
33 



■55 



OR 



127 
128 



53 



.15 



I 



90' 



60. 



a 



<58 



a 



89 



RESET S BIT / 61 



3 



ONLY COPY 



47. 



■ TRANSFER BLOCK 

-r. 

48 



REMOTE , l \ 



TRANSFER 

i • 1 — Ns 



BLOCK 
46 



CT 



TO SHAREO STORAGE 



59 



29 



T 



39-1 



40 



V S Ft 
L I i I I 



SET LOCAL S BIT 

r — 



COMPARE 

E 



56 



41- 



a 



REMOTE TRANSFER BLOCK 



~T~ 
31 



STORE 



REMOTE FETCH 



92, p 



28 



16 



98 



RESET LOCAL 
S BIT 



a 



FIG. 4 



99 



GT 



,47 



10 54 A i S STORAGE 



a 



94 

jjjp RESET REMOTE F B]T 



95 



RESET LOCAL V BIT 



96, 



93 



a 



SET LOCAL I REMOTE F 3IT ^ 



10/09/2003, EAST Version: 1.04.0000 



3,735,360 

1 2 

HIGH SPEED BUFFER OPERATION IN A the resetting of an associated valid bit and any subse- 

MULTI-PROCESSING SYSTEM quent fetches to the same Week require accessing the 

BACKGROUND OF THE INVENTION °T St ° l *f ^ fetch the date to the buffer. 

A second method is the "store wherever." In this 

1. Field of the Invention 5 method, the buffer address mechanism is checked to 
The invention relates to data processing systems and determine if the address block is currently in the buffer. 

more particularly to multi-processing systems wherein If the Week is in the buffer, the data is stored directly 

each processor employs a high-speed buffer or private into the buffer without further action. If the block is not 

store-in combination employs a storage device shared in the buffer, the data is stored in the main storage, 

by all processors. 10 The third method, "store-in buffer", for which the 

2. Prior Art present invention is primarily adapted, brings the block 
An article by C. J. Conti entitled "Concepts For from main storage and then stores the new data into the 

Buffer Storage" published in the IEEE Computer block in the buffer. 

Group News, March 1969, describes a hierarchical The above-mentioned Conti article discusses various 
memory in which a large slow speed three dimensional 15 techniques for organizing data and access to that data 
core storage operates in conjunction with a relatively in the high-speed buffer. One such technique, for which 
small high-speed buffer storage (or cache) manufac- the present invention is primarily adapted, is known as 
tured using integrated circuit technology. By using the the "set associative" technique. An example of this 
buffer/backing store arrangement, the central process- technique can be found in U. S. Pat No. 3,588,829, 
ing unit (CPU ) is able to access data at a high rate from 20 Ser. No. 776,858, Filed Nov. 14, 1968 and which is as- 
the high-speed buffer which is matched more closely to signed to the same assignee as this application. In this 
the machine cycle of the CPU. When the CPU provides technique, the address information is broken down into 
the address of desired information to the memory sys- books, pages and words. Depending on the memory 
tern, a control circuit determines whether or not the ad- size, there can be some predetermined number of 
dressed data has been moved from the backing store to 25 books, having a predetermined number of pages, and 
the buffer store. If the data is located in the buffer each page containing a predetermined number of 
store, high speed access is possible from the buffer words. As an example, it can be determined that each 
store to the CPU. If the data is not in the buffer store, book should contain 128 pages and that each page 
controls move the data from the backing store to the should contain some predetermined number of words, 
high-speed buffer and access is possible. A use algo- 30 When this determination is made, it specifies that the 
rithm is provided to insure that the most frequently high-speed buffer will have 128 storage sections, each 
used data is stored in the high-speed buffer. If the use section containing the number of words in a page. As- 
algorithm is efficient, most accesses will be to the sociated with each of the 128 sections of high-speed 
higher speed buffer store. This should result in a com- storage will be a directory, or address index array, con- 
tained system having effective speeds approaching that 35 taining 128 registers. In the set associative technique of 
of the fastest memory at a cost approaching that of the access, the corresponding page number from any of the 
slowest and least expensive memory. predetermined number of books will always be found 

In the prior art. buffer/backing storage apparatus are in the same storage section of the high-speed buffer, 
transparent to the user and the buffer operation is That is, page 10 from any book in the backing store will 
under fixed hardware control. When a CPU initiates a 40 always be found in location 10 in the high-speed buffer, 
fetch operation, the main storage address is presented The associated register in the directory will be provided 
to the memory hierarchy. Controls access a search with an entry which identifies the particular book to 
mechanism or directory of the high-speed buffer to de- which this particular page 10 belongs. The method of 
term ine if the requested address currently resides in the determining if requested data is in the high-speed 
high-speed buffer. If the requested information is in the 45 buffer is to utilize the address bits specifying pages to 
buffer, it is immediately made available to the CPU. If access the directory, and simultaneously therewith, ac- 
the requested information is not currently in the buffer, cess the high-speed buffer. The entry in register 10 of 
a fetch operation is initiated to the main storage back- the directory is compared with the applied address to 
ing store. The buffer location to receive the informs- determine whether or not the book value of the applied 
tion from main storage is determined by replacement 50 address matches the book value contained in the regis- 
logic which, in accordance with some predetermined ten If they do compare, this indicates that the re- 
algorithm, determines which address in the buffer store quested page 10 from the requested book is the data 
is to be replaced with the new data unit. When the fetch contained in the high-speed buffer. If the data is not 
is initiated at the main storage, the exact word re- from the requested book, the page 10 from the re- 
quested is first accessed and sent directly to the CPU " quested book is transferred from the backing store to 
and the buffer, followed by the remaining words in the the buffer and inserted in storage section 10 of the 
same block of data as determined by the particular high-speed buffer and the identity of the requested 
block size of the system. book is then inserted in the associated register of the 

There are currently three method* in the prior art for ^ directory, 
handling store operations. The "store through" method Another technique, for which consideration has al- 
ia used on most existing systems and the data is always ready been given in multi-processing systems, is known 
stored immediately In the main storage and the buffer as the "fully associative" technique. In this technique, 
address mechanism is checked to determine if the ad- the high-speed buffer may be, for example, provided 
dress block is currently in the buffer. If the block is in « with 16 storage sections. Associated with each storage 
the buffer, the data is also stored in the buffer. How- section will be a register. The size of each storage see- 
ever, on some systems, where I/O operations only store tion may be capable of storing an entire book. The pax- 
into main storage, the buffer block is made invalid by ticular book stored in a particular one of the storage 
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sections will be identified in the associated register. As dress information to all other processors from a partic- 
cach address is applied, the book address portion is ular one of the processors in addition to the access con- 
compared with the entries in all of the registers and if trol signal to indicate whether or not the operation is 
a match is found, the data is specified as being in the to be a fetch or a store. Utilizing the broadcasted ad- 
section associated with that register. In the fully asso- 5 dress and access control signals, the storage control 
ciative technique, data transferred from the backing mechanism of all other processors is operated to deter- 
store to the buffer store can be placed in any of the lo- mine further action in connection with the data re- 
cations. When new data must be inserted, a replace- quested by the particular processor, 
ment algorithm determines which of the sections The private high-speed buffers have a predetermined 
should be replaced and new data is inserted in that sec- 10 number of storage sections and an associated directory 
tion and the identity of the book is inserted in the asso- register for identifying the address of the shared storage 
ciated register of the directory. data presently stored in the high-speed buffer. By pro- 
One prior art technique has been shown in a multi- viding various combinations of additional binary bits in 
processing environment which utilizes a fully associa- the registers of the directory or index array, various 
tive configuration and the store-through concept. A 15 forms of storage organization and access methods can 
storage protect memory, which is utilized to protect a ^ controlled by the broadcasted address and access 
predetermined fixed amount of data in the backing control information to insure that each processor ac- 
store, is provided with additional binary bits for reflect- cesses the most current value of the operand identified 
ing which of the several processors has accessed data in ^ shared storage. 

from the backing store to its associated private store. 20 tf on]y Qnc ^^,^1 con trol bit is provided in each 
Whenever a processor stores data into the backing of the directory registers, which sigriifies the validity of 
store, utilizing the store-through concept, the storage ^ m ^ associated storage section, each proces- 
protect memory is interrogated and if it is determined ^ must broadcast address and access control informa- 
that the data block is in another processor's private tion whenever data is to be stored by a particular pro- 
storage, the mechanism will be utilized to invalidate the 25 cessor Tne ^rectory of all other processors is searched 
data in the other processor requiring that processor to tQ determine whether or not the same data is contained 
fetch the data from the backing store the next tune it - n ne private storage . if so, the validity bit is 
is utilized. This prior art technique is limited to a buffer feset to reflcct ^ ^ daU b nQ Iongef va|id in Ac ^ 
storage configuration in wmch each storage section pri vate storage. Another bit which can be pro- 
must contain the same amount of data as specked in *> yided m ^ direcl registers is a bit called a fetch- 
the storage protect memory, and does not address itself onj ^ ^ W| u ^ ^ reset to reflect whether Qr 
to a set associative configuration nor does it consider not ^ ^ fa ^ one of ^ rivate storages 
the problems arising when utilizing the store-in buffer ^ ^ Q ^ y copy of ^ ^ - n a prWatc stQrage 
concept. 35 Tnal ^ tf a particular processor has fetched data from 
BRIEF DESCRIPTION OF THE INVENTION the shared storage into the private storage, and it is 

t.. ^ • *• • j t j n . known that this is the only copy of the data in the pri- 

It is an object of this invention to provide a broadcast * \i ... , # . . -J- 

. , J , . . 1 vate storages, the fetch-only bit will reflect this. The 

or .ntcrlock mechan.sm between a plurahty ofdrt. for ? roadcasting ^ ^dress and access control 

processors each havmg a h.gh -spee, pnw te sto^e ^ 0 ^ on would not exist . Another 

and each accessing data from a large shared storage. w *• . *7 . . . . . _ ... 

It is also an object of this invention to provide high- binary bit which can be provided in the registers of the 
speed buffer operation in a multi-processing configura- * re <*^ or ind « ^ » » to ^own as a store bit. 
tion to insure Uiat each processor accesses the most ?™ b " is set and reset ^ reflect a condition wherein 
current value of a particular operand. ^ * e »> * e h,^-speed buffer of a particular proces- 

It is another object of this invention to provide high- 45 sor data m the shared storage. That is, 

speed buffer operation in a multi-processing configura- wh * n ut *™S the store-in buffer concept, all accesses 
tion wherein the invention can be adapted to various to data by particular processors are made in the high- 
storage control techniques such as store-in-buffer, speed buffer including accesses for the storage of data 
store-through, fully associative access, or set- ^ When the data has been transferred to the high-speed 
associative access buffer of a particular processor, and that data is subse- 

The above objects are accomplished in a multi- quently stored into in the buffer, the store bit is set. 
processing system which includes a shared storage and Whenever a particular processor request requires 
a plurality of processing units. Each processing unit in- transfer of new data from the shared storage to the 
eludes a private, high-speed buffer storage, an associ- „ high-speed buffer for either storing or fetching, the ad- 
ated directory for providing an indication of the data M dress and access control signals will be broadcast to the 
transferred from the shared storage to the high-speed other processors. The address information of the re- 
buffer, and a storage control means which accepts sig- quested data is utilized to search the directories of all 
nals from the associated processor, including the other processors to determine whether or not the re- 
shared storage address of data to be operated on, and ^ quested data resides in one of the other private high- 
an accessing control signal which indicates that the speed storages and whether or not that data has been 
data is to be fetched for transfer to the processor or stored into. If the data has been stored into by another 
that the processor is to store data into the operand lo- processor, that data must first be transferred back to 
cation. the shared storage in its modified form so that the pro- 

The present invention provides means interconnect- 65 cessor requesting the data will receive from the shared 
ing all the processors to perform an interlocking func- storage the most current value. This requirement is not 
tion. The interlocking function is accomplished by necessary if the determination is made that the data in 
broadcasting, under certain specified conditions, ad- the other processor has not been stored into and there- 
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fore has the same values as the operands in the shared of the present invention in which various controls are 
storage. energized in dependence on whether or not more than 

BRIEF DESCRIPTION OF THE DRAWINGS ° nC Copyofa P«*»l* block <* operands exists in the 

various private storages. 

FIG. 1 is a block diagram showing the interconnec- 5 FIG. 2 is a flow chart of the logic decisions and se- 

tion for broadcast purposes between processors each quences of decisions made in response to a request for 

having private high-speed storage. access to a shared storage location by a processor, 

FIG. 2 is a flow chart of logic decisions and se- wherein the access request is for the purpose of fetch- 

quences. ing data or storing data in the accessed location. Before 

FIG. 3 is a logic diagram showing the basic controls 10 discussing the sequences as shown in FIG. 2. a brief de- 

of a storage control unit in each processor and the logic scription of the general makeup of the private storage, 

for determining the need for broadcasting information. directory, and storage control apparatus for one of the 

FIG. 4 is a logic diagram of the storage control unit processors will be discussed in connection with FIG. 3, 

in each processor which responds to broadcast address In FIG. 3, structure already discussed in connection 

and access control signals from a remote processor. 1 5 with FIG. 1 has been given the same numerical designa- 

DETAILED DESCRIPTION ti ° n * The P referred embodiment of the present inven- 

tion is utilized in a high-speed private storage system 
FIG. 1 shows generally the environment of the pres- wherein the set-associative method of ordering and 
ent invention. Operands to be utilized in the system are storing data is utilized along with the access method 
contained in a shared main storage 10. The operands 20 known as store-in-buffer. That is, every access request 
are accessed by a plurality of data processors 11 and by a processor must eventually be accomplished in the 
12. Each of the processors II and 12 identify operands high-speed storage, whether for the purposes of fetch- 
in shared storage 10 on address busses 13 and 14. Pro- ing data or storing data. 

cessors 1 1 and 12 have private high-speed storage 15 The private storage 15 is shown to include 128 stor- 
and 16 and data busses 17 and 18 for the transfer of 25 age sections 33. Each of the storage sections 33 has a 
data between processors and the local private storage. capacity for storing a block of data operands desig- 
A request for access to locations of operands specified nated as a page in the above-mentioned U. S. Pat. No. 
on the address busses 13 or 14 are signalled on access 3,588,829. Associated with each of the 128 storage 
control lines 19 and 20. The access control signals will sections 33, are 128 registers 34 forming the directory 
specify that the processor desires access to the operand 30 27. In accordance with the above-mentioned patent, 
location for the purpose of fetching data to the proces- one section 35 of each of the registers 34 will contain 
sor or storing data from the processor into the accessed the address designation of a particular book from the 
location. shared main storage 10. In other words, page 4 from 

The address information provided on busses 13 and any book in the shared main storage 10 will always be 
14 is applied to local storage control units 21 and 22 for 35 transferred to and stored in storage section number 4. 
the purpose of determining whether or not the data re- The particular book from which the page 4 was trans- 
quested is accessible in private storage 15 or 16. If the ferred will be designated in the section 35 of register 
requested data is in the private storage 15 or 16, the number 4. 

data will be immediately transferred on data busses 17 When an access request is signalled on line 19 from 
or 18. If the storage control unit 21 or 22 determines 40 the local processor ll t the local address information on 
that the requested data is not in the private storages 15 bus 13 will be passed through an OR circuit 36 for the 
or 16 respectively, a request will be made on control purpose of searching the directory 27 to determine 
lines 23 or 24 to initiate transfer of the data from whether or not the requested data is in the private stor- 
shared storage 10 to private storage 15 or 16 on storage age 15. The portion of the address information which 
data busses 25 or 26. The method of determining 45 specifies a page number will be utilized on busses 37 
whether or not the requested data is in the local private and 38 to access the designated register 34 and storage 
storage is by means of a search mechanism which in- section 33. The book address information will be read 
eludes directories 27 and 28. from the accessed register 34 and will be utilized in a 

In accordance with the present invention, the proces- compare circuit 39 to determine whether or not the 
son? are interconnected for the purpose of broadcasting block address information stored in the accessed regis- 
information necessary to insure that each processor ter 34 is equal to the block address information pro- 
will access operand locations which have the most cur- vided on the address bus 13. 

rent value of an operand in view of the fact that each The purpose of a number of additional binary bits as- 
of the processors, independently, may be modifying the sociated with each of the registers 34 will be more thor- 
operand values. Although various modifications to the oughly discussed subsequently. At present however, 
general concept of broadcasting will be discussed, the the presence of a valid bit 40 will be mentioned. When 
minimum amount of interconnections will include a the valid bit has a binary one value, and the compare 
bus 29 for transferring address information between circuit 39 indicates that the block address requested on 
the processors, and a control line 30 for signalling from ^ bus 13 matches the block address accessed in the regis- 
one processor to others that the one processor is ac- ter 34, an AND circuit 41 will provide an output signal 
cessing an operand location for the purpose of fetching on line 42 indicating a block-valid condition. That is, 
or storing data. In accordance with one modification the requested block of data is stored in the private stor- 
which specifies the store-in-buffer technique, another age 15 and is valid. 

interconnecting signal line 31 is provided for signalling M The address information provided on the bus 37 to 
from one processor to the others that a transfer is tak- the private storage 15 will access the identified storage 
ing place from shared storage to a private storage. In- section 33 and provide that data on a bus 43. In re- 
terconnecting signal line 32 is provided in another form sponse to an access request for fetching on signal line 
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19, and the determination that the block is valid in the being replaced to the shared storage on a bus 59 

private storage, an AND circuit 44 will provide a signal through a gate 60 enabled by the output of AND circuit 

to a gate 45 for the purpose of transferring the re- 58. When the data in the particular storage section is 

quested data immediately to the CPU on a bus 46. transferred back to the shared main storage, and the 

When, in response to the searching of the directory 5 new data transferred from main storage to the private 

27 with the address information on bus 13, it is deter- storage, the line 61 will be utilized to reset the store bit 

mined that the requested block of data is not validly 56 to binary zero reflecting that the data now contained 

stored in the private storage 15, an inverter circuit 47 in the storage section 33 is the same as that found in the 

will provide an output signal 48 indication the need to shared storage 10. 

transfer the requested block of data from the shared 10 One additional binary bit associated with each regis- 
storage 10 to the private storage 15. ter 34 of FIG. 3 will now be defined. That additional bi- 
If the private store and directory are configured in nary bit is referred to as the fetch-only bit 62. When 
accordance with the above-mentioned patent, a re- this fetch-only bit is a binary 0, it indicates to the stor- 
placement algorithm will be enabled to select a storage age control mechanism that this particular private s tor- 
section to receive the requested data. The address of 15 age has the only copy of the block of data from the 
the storage section to be replaced win be indicated on shared storage 10. That is, no other private storage 15 
a bus 45 which is also applied through OR circuit 36 to has requested this particular block of data. When the 
provide access to the register associated with the stor- fetch-only bit is in the binary I state, this indicates that 
age section to be replaced. The valid bit 40 associated some other processor has at some time transferred the 
with that register will be reset to indicate that the data 20 same block of data from the shared storage 10 to its pri- 
presently contained in the private storage 15 is no vate storage. 

longer valid. Further, the block identifying address por- The three most pertinent states of the valid bit 41 

tion of the requested data will be inserted into the ac- (V), store bit 56 (S), and fetch-only bit 62 ( F) is shown 

cessed register 34 on a bus 50. The block of data which in directory positions 1, 2, and 3, The state in position 

is returned from the shared main storage 10 will be on 25 i indicates that this processor's private storage con- 

a bus 51 applied through a gate 52 and OR circuit 53 tains the only copy of the identified block of data. This 

to the storage section selected for replacement. particular block can be stored into by this processor 

If the requested block of data which was transferred without affecting the same data in any other private 
from the shared main storage to the private storage was storage. The state indicated in position 2 indicates that 
in response to a fetch access request by the associated 30 the block is valid in this particular private storage but 
processor, the AND circuit 44 will now provide an indi- that it also exists (or did exist at some time ) in another 
cation necessary to energize gate 45 to transfer the re- processor's private storage. This particular processor 
quested operand to the processor on bus 46. To be can only read data from this block without the require- 
more fully discussed subsequently, if the requested ment for notifying another processor of any action. Be- 
block of data was to be brought to the private storage 33 fore the processor can store into this block, a broadcast 
15 for the purpose of storing data in one of the operand of information must be made to invalidate the data in 
locations, the data to be stored into the private storage the other private storages and change the designation 
will be provided on a bus 54 through an enabled gate in this private storage to that shown in position 1. The 
55 and the OR circuit 53 to the identified operand loca- state indicated in position 3 is essentially the same as 
tion in the storage section 33. 40 that shown in position 1 except that this block of data 

When it is determined that a block of data in one of - has been stored into by this processor and therefore is 
the storage sections 33 of private storage 15 is to be re- the most up-to-date copy of this block of data, 
placed, one additional binary bit associated with each Discussion will now return to FIG. 2 to provide a gen- 
of the registers 34 will be effective. The relationship of eral indication of logic decisions and sequences which 
this additional bit, labeled a store bit 56 will be more must be made in order to cause all of the private stor- 
thoroughly discussed in connection with the broadcast ages of all processors to reflect the correct value of a 
mechanism. In can be utilized to indicate that the data particular operand in view of the fact that each proces- 
to be replaced in the selected storage section 33 has sor may be operating independently with the data con- 
been modified or stored into by the associated proces- tained in its associated private storage. In FIG. 2, the 
sor while in the storage section 33. Whenever an asso- designation B-l designates the requested block of data 
ciated processor stores data into storage section 33, the by the associated processor. The designation B-2 indi- 
store bit 56 in the associated register 34 will be set to cates the block of data in a private storage which is to 
a binary one condition. When the indication for a data be replaced by new data. 

transfer is given on line 48, a further signal indicating 53 In response to a fetch or store access request from 

the possible need to restore a block will be given on a processor A, decision block 63 will determine if the 

signal line 57. AND circuit 58 will make the determina- block- valid signal is produced for the requested block 

tion that the data in the storage section 33 to be re- in buffer A. If the block is valid, decision 64 will deter- 

placed is valid and has been stored into. The need for mine whether or not it is a fetch request or a store re- 

the store bit is more evident when h is recalled that the ^ quest. If a fetch request, the action taken at 65 will fol- 

store-in -buffer concept is utilized. The store bit 56 hav- low. The data from the requested block B-l of buffer 

ing a binary one condition indicates that the data in the A is returned to the processor A. When decision block 

storage section 33 of the private storage 15 has been 64 determines that the request is for a store operation, 

modified and is no longer identical to he same block of decision block 66 will determine whether or not the 

data retained in the shared main storage 10. Therefore, 65 fetch-only bit is on or off for the requested block in 

when the data in the private storage differs from the buffer A. If the fetch-only bit is off, the action shown 

data retained in the shared storage, AND circuit 58 will at 68 will take place. Namely, the data from processor 

be utilized to initiate the transfer of the block of data A will be stored into the proper operand location of 
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block B-l in buffer A. Also, the action of storing into indicate that the data in shared storage 10 is now the 

block B-l of buffer A will cause the store bit to be same as the data found in the buffer of processor B. 

turned on in buffer A. When the block B-l has been restored to shared stor- 

lf decision 66 indicates that the fetch-only bit was a age 10, or it has been determined that this is not re- 
binary 1 , this indicates that other private storages con- 5 quired, the next determination shown at 77 is whether 
tain (or did contain at some time) a copy of the same or not the access request at processor A is for the pur- 
block of data. Therefore, the need for broadcasting in- pose of fetching data or storing data. If the access re- 
formation on the interconnecting means between pro- quest at processor A is not for a fetch, and therefore a 
cessors is initiated. The basic information broadcast is store, the action taken at 78 is to turn off the block 
the address of the requested block B-l and whether or 10 valid bit for block B-l in the buffer of processor B 
not it was for a fetch or store access request. When the thereby forcing processor B to make its next request for 
broadcast data is received at the other processors, deci- an operand from block B-l to shared storage 10. 
sion block 69 will determine whether or not the re- If the decision at 77 indicates that the request at pro- 
quested block B-l is valid in that particular private cessor A is for the purpose of fetching data, the fetch - 
storage, herein designated processor B. If the requested 15 only bit for the block B-l in buffer B is turned on at 79 
block B-l is not valid in the other private storage, the and the fetch-only bit for block B-l in processor A is 
fetch-only bit in the buffer of processor A will be turned on at SO thereby reflecting that more than one 
turned off at 67 and the store operation can take place copy of block B-l exists in the private storages of all 
at 68. processors. 

When it is determined that the requested block B-l 20 If as a result of the broadcast of information, the de- 
is valid in the private storage of processor B, the block cision is made at 74 that the requested block B-l is not 
valid bit for the storage section containing the re- validly in the buffer of processor B, the fetch-only bit 
quested block B-l will be turned off at 70 since the for the requested block B-l in the buffer of processor 
broadcast was the result of a store access request in A will be turned off at 81 reflecting that the buffer of 
processor A. This will have the effect of causing pro- 25 processor A has the only copy of block B-l other than 
cessor B to request a transfer of the data from the that found in the shared storage 10. 
shared storage 10 to its private storage the next time When it has been determined that the block which 
processor B attempts access of the data in block B-l. must be transferred from shared storage 10 to the 
When the block valid trigger has been turned off for buffer of processor A is valid in the shared storage 10, 
block B-l in buffer B, the fetch-only bit for block B-l 30 the block B-l will be transferred from the shared stor- 
in processor A will be turned off at 67 and the store op- age 10 to the selected storage section of the buffer of 
eration can take place at 68. processor A and the valid bit in the associated register 

The remainder of the logic decisions and sequences for block B-l will be turned on. This action is shown at 

shown in FIG. 2 take place when it is determined at 63 82. When the data has been transferred from shared 

that the requested block B-l is not valid in processor 35 storage 10 to the buffer of processor A. the determina- 

A. When the requested block is not valid in buffer A, don of a fetch or store request is made at 83 and the 
the replacement algorithm is enabled at 71 to pick a actions indicated at 65 or 68 will take place. 

block to be replaced in buffer A, and will subsequently The logic decisions and sequences discussed in con- 
be identified as block B-2. At this point, the decision is nection with FIG. 2 will now be related to FIGS. 3 and 
made at 72 as to the need for restoring the data from 40 4. FIG. 3 is intended to represent that portion of logic 
the private storage back to the shared main storage 10. necessary for one of the processors to initiate a broad- 
As indicated previously, this decision depends on the cast, or transfer, of access control information and ad- 
condition of the valid bit and store bit in block B-2 of dress information on the interconnecting means. FIG. 
the buffer of processor A. If the valid and store bit are 4 shows the logic required in other processors for re- 4 
on, the action at 73 takes place. Namely, the block B-2 45 sponding to the broadcast information, 
to be replaced is transferred to the shared storage 10 The need for broadcasting address information on 
from buffer A and the store bit for the storage section the interconnecting address bus 29 and the transfer of 
which contains B-2 in buffer A is turned off. When the the access control signal on line 30, to be considered 
restoring of the block of data has taken place at 73, or as remote signals, is accomplished by an OR circuit 84, 
it is determined that it is not needed at 72, broadcasting gate 85, and gate 86. The need to broadcast address 
of address and access control information must take and access control information based on the decisions 
place. The need for the broadcast of information at this of FIG. 2 indicating that the requested block is valid in 
point is to determine whether or not the requested the requesting system and that the access is for the pur- 
block B-l is contained in the buffer of processor B and pose of storing information is represented by an AND 
whether or not the value of the operands in the buffer circuit 87* AND circuit 87 responds to the block valid 
or processor B are the same as, or different from, the signal from AND circuit 41, an indication that the 
block of operands in shared storage 10. fetch-only bit for the requested block is a binary 1 and 
The broadcast address and access control signal is the signal that the access request is a store operation 
utilized to search the directory in processor B for the ^ generated from inverter 88. The output of AND circuit 
presence of the requested block B-l, and the decision 87 is applied to OR circuit 84 to thereby energize gates 
as to whether or not block B-l is valid in buffer B is de- 85 and 86 to broadcast, or transfer on the interconnect- 
termined at 74. If the requested block B-l is in buffer ing means, the requested block address and the access 

B, and the store bit for the requested block B-l in request As mentioned earlier, if the fetch bit 62 for the 
buffer Bis one as indicated at 75, the block of data B-l w requested block is binary 0, indicating that this is the 
must be restored to shared storage 10 from the buffer only copy of the data, AND circuit 87 will not produce 
of processor B as shown at 76. Also, the store bit for an output signal and will therefore inhibit the broad- 
block B-l in the buffer of processor B is turned off to casting of information. 
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As discussed in FIG. 2, when the processor request- 
ing information detects that there is a need for transfer- 
ring the block from the shared storage 10 to the private 
storage 15, the signal on line 48 indicating a need to 
transfer a block is applied to OR circuit 84 to thereby 
enable gates 85 and 86. The signal on line 48 is trans- 
ferred as a remote signal to other processors to initiate 
the decisions starting at 74 in FIG. 2. 

Other logic shown m FIG. 3, which responds to the 
initial search of the directory 27 by the applied local 
address on address bus 13, includes an AND circuit 89 
which responds to a block valid signal and the require- 
ment of a store access request to set the S bit 56 associ- 
ated with the accessed storage section and register. In- 
verter 90 and AND circuit 91 respond to a search of 
the directory 27 to indicate that the requested block is 
valid and that it is the only copy of the requested block 
of data. 

Referring now to FIG. 4 V there is shown the logic in 
all of the processors which is rendered effective when 
information is broadcast or transferred on the intercon- 
necting address bus 29 and access control line 30. The 
only additional line required to be transferred on the 
interconnecting means to other processors is the line 
labeled 31 signifying that the broadcasting processor is 
required to transfer a block of data from the shared 
main storage 10 to the private storage. The broadcast 
of address information will be utilized to search the di- 
rectories of other processors. 

In FIG. 4, the directory 28 of processor B and private 
storage 16 of processor B is shown. The same compare 
circuit 39 and AND circuit 41 will provide the block 
valid signal on line 42 and a block not valid signal from 
an inverter 47. An inverter 92 responds to the remote 
access request line 30 to indicate when a remote store 
is taking place. AND circuit 93 provides the decision 
indicated in decision block 69 of FIG. 2. When the re- 
quested block is valid in the other processors, and the 
processor which is broadcasting is storing information, 
AND circuit 93 will be effective to reset the valid bit 
40 of the corresponding block of data in processor B 
which is being stored into by processor A. At the same 
time, the output of AND circuit 93 will be effective at 
OR circuit 94 to transfer to processor A on the inter- 
connecting means, on line 95, the signal necessary to 
reset the fetch-only bit 62 of processor A to reflect that 
processor A now has the only valid copy of the block 
of operands for storing into. OR circuit 94 also re- 
sponds to inverter 47 which signals that the block re- 
quested by processor A is not valid in the private stor- 
age of processor B to also thereby reset the fetch-only 
bit of processor A. 

An AND circuit 96 responds to the remote fetch sig- 
nal 30 and block valid signal from AND circuit 41 to 
indicate both to the local directory 28 of processor B 
and the directory 27 of processor A that more than one 
copy of the requested block of data exists in the private 
storages. This line labeled 97 sets the local F bit and is 
effective on the interconnecting means to set the F bit 
of processor A. 

The remaining logic shown in FIG. 4, AND circuit 
98, provides the decision shown at 72 of FIG. 2. That 
is, when processor A has signalled that it is transferring 
a block of data on line 31, that the requested block of 
data is valid in processor B, as signalled on line 42, and 
processor B has stored into the block of data as indi- 
cated by the binary 1 condition of the store bit 56, the 
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contents of the storage section of private storage 16 
will be transferred by a gate 99 to its proper location 
in the shared main storage 10. Also, the output of AND 
circuit 98 will be utilized to reset the local S bit 56 to 
5 reflect that the value of the operands transferred to the 
shared main storage 10 are now identical to the data 
contained in the storage section of the private storage 
16. 

Returning now to FIG. 3, the remainder of the logic 

10 shown will be discussed. The indication that the local 
processor is storing information into a block of data 
which is the only copy outside of shared storage 10 is 
indicated by an AND circuit 100, and an OR circuit 
101. The output of AND circuit 100 will be effective 

1 5 at gate 55 to immediately transfer the data on bus 54 
from the local CPU into the accessed storage section 
33 of private storage 15. The other input to OR circuit 
101 is provided by the interconnecting signal line 95 
indicating that the other processors have reset the 

20 fetch-only bit 62 in the broadcasting processors direc- 
tory. AND circuits 102 and 103 will be rendered effec- 
tive when inverter 47 indicates a need to transfer a 
block of data from the shared main storage 10 to the 
local private storage 15. Gate 52 which transfers the 

25 data on bus 51 from shared storage to private storage 
15 will be enabled through an OR circuit 104. 

The direct application of the reset remote F bit signal 
line 95 to OR circuit 104 reflects the decision made at 
74 in FIG. 2 and is generated in response to the deter- 

30 mination that the requested block is not contained in 
any other private storage. AND circuit 102 reflects the 
decision made when the local processor wishes to store 
data into a block, but that block has to be transferred 
from the shared main storage 10 to the local private 

35 storage 15. When the need for a block transfer from 
shared storage 10 to private storage 15 is signalled, the 
action block 78 of FIG. 2 reflects that the valid copy in 
processor B is made invalid by the AND circuit 93 of 
FIG. 4 which also generates, through OR circuit 94, the 

40 reset remote F bit signal 95. When this has been re- 
ceived by AND circuit 102, OR circuit 104 will enable 
gate 52 to transfer the block of data from the shared 
storage 10 to the private storage 15. 
AND circuit 103 reflects the decisions made which 

45 ultimately generates the signal shown in action block 
80 of FIG. 2 which turns on the fetch-only bit in the pri- 
vate storages of both processors. Once again, OR cir- 
cuit 104 provides the indication to initiate the transfer 
of a block of data from shared storage 10 through gate 
52* A delay circuit 105 generates a signal to set the 
valid bit 40 in the directory 27 after the block of data 
has been transferred to the selected storage section 33 
of the private storage 15* 

55 As mentioned previously, the preferred embodiment 
of the present invention includes a private storage and 
directory configuration utilizing the set-associative 
technique. Further, the storage method known as store- 
in-buffer is implemented, and various controls and de- 

£Q cisions are generated in response to the valid bit, store 
bit, and fetch-only bit. Various modifications can be 
made to this basic system. The directories 27 or 28 may 
contain only a valid bit 40. In this situation, whether 
store-in-buffer, store through, or store wherever is uti- 

65 lized, the need for the broadcasting of address and ac- 
cess control information is required whenever a storage 
operation into a private store or shared storage is ac- 
complished. The dotted control line 106 of FIG. 3 re- 
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fleets this situation. That is, whenever a processor 
stores information, the other processors must be inter- 
rogated with the broadcast address and access control 
information to invalidate the data in any other private 
storage which also contains the block of data being 5 
stored into. 

The next possible modification is to add to the previ- 
ously mentioned valid bit 40, the fetch-only bit 62 
which would negate the need to broadcast this informa- 
tion on a store operation when it is determined that the 10 
block of data being stored into is only contained in a 
single private storage. When using only the valid bit or 
the valid bit and the fetch-only bit, and when there is 
a need to transfer a block of data from the shared main 
storage 10 to a requesting processor, there will be a 15 
need to determine whether or not block of data resides 
in any other private storage. If the block of data does 
reside in another private storage, it will be necessary to 
initiate a transfer of the block of data from the other 
private storage to the shared main storage 10 prior to 20 
transferring the block to the requesting processor. Fur- 
ther, any block being replaced in a particular one of the 
private storages will always have to be transferred back 
to its proper location in the shared main storage 10 
since it will not be known for certain whether or not 25 
that data has been modified while in the local private 
storage. 

By the addition to each of the registers in the directo- 
ries of the store bit 56, the need for initiating a transfer 
of blocks of data from a private storage to the shared 30 
main storage can be eliminated when it is determined 
that the block of data in the private storage has not 
been stored into prior to the time it is replaced by the 
replacement algorithm. 

Although the preferred embodiment of the present ^ 
invention is utilized in a set-associative configuration, 
the fully associative technique can be utilized. By the 
provision of the additional control bits in each of the 
associative registers, the various storage control meth- 
ods can be implemented. Further, by associating the 40 
valid bit 40, store bit 56, or fetch-only bit 62* more flex- 
ibility is provided in choice of the size of the block of 
data transferred back and forth between private stor- 
age and the shared storage. By eliminating the need to 
equate the necessary interlocks to a predetermined 
block size which is protected by another mechanism, 
there would not be a need to invalidate the entry in an- 
other private storage whenever any one particular op- 
erand is modified out of the block of protected oper- 
ands. 

While the invention has been particularly shown and 
described with reference to a preferred embodiment 
thereof, where interconnecting means are provided be- 
tween a plurality of processors sharing a main storage 55 
so that each processor can operate with a private high* 
speed storage and maintain access to the most current 
value of any particular operand referenced in the 
shared main storage, it will be understood by those 
skilled in the art that various other changes in form and ^ 
details may be made therein without departing from the 
spirit and scope of the invention. 

What is claimed is: 

1, A data processing system comprising: 

shared storage means for storing a plurality of oper- 6 ^ 

ands at addressable locations; 
a plurality of processing means, each including 

means to provide a local address signal identifying 
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an operand location in said shared storage means, 
and local access control means for signalling an ac- 
cess request for fetching data from or storing data 
in the addressed location; 

each said processing means having connected thereto 
a local high speed buffer system including : 

private storage means connected to said shared stor- 
age means for storing a predetermined portion of 
operands previously transferred from said shared 
storage means to said private storage means, 

directory means for identifying the operands in said 
private storage means for immediate access by said 
processor and, 

storage control means including means responsive to 
said local address signal means, said local access 
control means, and said directory means for pro- 
viding in said private storage means, access to an 
operand from an identified operand location; and 

means interconnecting all of said processing means 
and said high speed buffer system responsive to ad- 
dress signals provided by said processing means 
representing a particular operand location for 
causing the most current value of the particular op- 
erand to be accessed by all said processing means. 

2. A data processing system in accordance with claim 
1 wherein, 

each said private storage means includes: 
a plurality of storage sections, each said section 
storing a block of a predetermined number of op- 
erands transferred from said shared storage; 

each said directory means, includes: 
a plurality of registers each of said registers being 
associated with a predetermined one of said stor- 
age sections, and each including a block address 
portion and valid bit having first and second 
states for identifying the block of said shared 
storage operands in said storage section and the 
validity thereof when said validity bit is in said 
first state; 

each said storage control means includes: 
search means, responsive to said local address sig- 
nals, including means for searching said directory 
and providing a block-valid signal or block-not- 
valid signal dependent on whether or not the ap- 
plied local block address identifies a block with 
valid data in one of said storage sections, and in- 
cluding processor data gating means connected 
between said local private storage and processing 
means and responsive to said block-valid signal 
and said local access control means for providing 
access by said processor to the identified oper- 
and in said storage section; and 
said interconnecting means includes: 
broadcast means in each of said processing means, 
including remote signalling means connected and 
responsive to said local address signals and said 
local access request control signal for storing 
data for transferring said signals from any one of 
said processing means to said search means of 
other of said processing means; and 
means in the other of said processing means re- 
sponsive to said block-valid signal and said re- 
mote access request control signal for storing 
data to generate an invalidate signal to change 
said valid bit to said second state in the one of 
said registers having said block address portion 
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the same as the block address of said remote ad- 
dress signals. 

3. A data processing system in accordance with claim 

2 wherein, 

each of said registers further includes: 5 
a fetch-only bit having a first or second state, the 
first state indicating that the block of operands 
transferred from said shared storage to said asso- 
ciated storage section is valid in one of said pri- 
vate storage means of said other of said process- 10 
ing means; and 
said broadcast means of said interconnecting means 
further includes: 

means connected and responsive to the first state 
of said fetch -only bit, whereby said interconnect- 1 5 
ing means is enabled only when said block of op- 
erands is validly stored in more than one of said 
private storage means. 

4. A data processing system in accordance with claim 

3 wherein, 20 
said interconnecting means further includes: 

reset signalling means in said other of said process- 
ing means, connected and responsive to said 
block-not valid signal or said invalidate signal for 
resetting said fetch-only bit in said register of said 25 
one of said processing means. 
5* A data processing system in accordance with claim 
2 wherein, 

said remote signalling means of said broadcast means 
further includes: 30 
means responsive to said block-not valid signal in 
said one of said processors for transferring said 
applied local address and said block-not valid sig- 
nal to said search means of said other of said pro- 
cessors; 35 
said search means of said other of said processors fur- 
ther includes: 

up-date gating means, responsive to said block- 
valid signal and said remote block-not valid sig- 
nal and connected to said storage data gating 40 
means for transferring the block of operands 
identified by said remote block address from said 
storage section to said shared storage; and 
said search means of said one of said processors fur- 
ther includes: 45 
storage data gating means connected between said 
local private storage and said shared storage re- 
sponsive to said block-not valid signal, for select- 
ing one of said storage sections, and for transfer- 
ring the block of operands from said selected SO 
storage section to said shared storage and the 
block of operands identified by the applied local 
block address from said shared storage to said se- 
lected storage section, and for entering the block 
address in said associated register and for setting 55 
said valid bit. 
6* A data processing system in accordance with claim 
5 wherein, 

each register in each of said directory means in- 
cludes: 60 



a store bit having first and second states, said first 
state indicating that the block of operands in said 
associated storage section has been stored into 
by said local processor, 

said up-date gating means is further responsive to the 
first state of said store bit; and 

said storage data gating means from said private stor- 
age means to said shared storage is further respon- 
sive to the first state of said store bit and said valid 
bit of said register associated with said selected 
storage section, 

whereby transfer of blocks of operands from said pri- 
vate storage to said shared storage is only effected 
when the block of operands in said private storage 
has been stored into and therefore differs from the 
operands in said shared storage. 

7. A data processing system in accordance with claim 
6 wherein, 

each said storage control further includes: 
means for resetting said store bit of said registers 
when the block of operands of said associated 
storage section are transferred to said shared 
storage. 

8. A data processing system comprising: 

shared storage means for storing a plurality of oper- 
ands at addressable locations; 

a plurality of processing means, each including 
means to provide local address signals identifying 
an operand location in said shared storage means, 
and local access control means for signalling an ac- 
cess request for fetching data from or storing data 
in the addressed location; 

each said processing means having connected thereto 
a local high speed buffer system including : 

set-associative private storage means connected to 
said shared storage means for storing a predeter- 
mined portion of operands previously transferred 
from said shared storage means to said private stor- 
age means, 

directory means for identifying the operands in said 
private storage means for immediate access by said 
associated processor and, 

storage control means including means responsive to 
said local address signal means, said directory 
means and said local access control means for pro- 
viding in said private storage means, access to an 
operand from an identified operand location in re- 
sponse to a fetch request, and, in response to a 
store request, providing access to said identified 
location in said shared storage and said private 
storage upon condition that said identified operand 
is in said private storage; and 

means interconnecting all of said processing means 
and said high speed buffer system, responsive to 
address signals provided by said processing means 
representing a particular operand location for 
causing the most current value of the particular op- 
erand to be accessed by all said processing means. 
***** 
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